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1 Introduction 

On November 18, 1999, the NASA/Ames Independent Verification and 
Validation (IV&V) Facility presented their Annual Review Presentation to the 
Office of Safety and Mission Assurance (OSMA) in Washington, DC. This 
meeting was designed to report on IV&V Facility accomplishments achieved 
through activities funded by OSMA. The meeting was originally scheduled for 
9:00 a.m. to 12:00 noon and was shortened due to scheduling conflicts. The 
actual time for the meeting was approximately 9:15 a.m. to 11:15 a.m. 

Several questions and concerns were brought forth during the course of the 
meeting. This report contains recommendations for addressing the action 
items requested as well as some of the issues that arose. 


Attendee 

Organization 

Dr. Louis Blazy 

NASA IV&V Facility 

Cynthia Calhoun* 

NASA IV&V Facility 

Ken Costello (telecon) 

NASA IV&V Facility 

Marcus Fisher (telecon) 

NASA IV&V Facility 

Dr. Michael Greenfield 

NASA OSMA 

Melanie Gould* 

D.N. American/ SETA Contract 

Frank Huy (telecon) 

D.N. American/ SETA Contract 

Tim Menzies* 

West Virginia University 

Phil Napala 

NASA OSMA 

Pam Richardson 

NASA OSMA 

Edward Weed (telecon) 

D.N. American/ SETA Contract 

Siamak Yassini* 

NASA IV&V Facility 


* Indicates presenter. 
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2 Center Initiative Overview 

Siamak Yassini began the IV&V Facility’s OSMA Annual Review presentation 
with a summary of the budgetary and schedule status of the OSMA Software 
Program activities. 

2.1 Issues 

Concern 

The layout of the “Focus Areas” for the Center Initiatives (CIs) raised several 
questions from Dr. Greenfield. These included concern over whether all CIs 
were considered to be under “software assurance,” since it was not called out 
as a separate category, and whether or not each Cl would be described in detail 
since they were categorized by theme on three introductory slides. 

Analysis 

The IV&V Facility has historically spent a great deal of time on making 
presentation packets audience specific and concise, thus getting to the “meat” 
of the information as quickly as possible. In this briefing, the CIs are 
accompanied by several introductory slides showing status, NASA Enterprise 
penetration, and funding. This inevitably leads to some repetitive material. 

Recommendation 

It is recommended that the thematic division of the CIs be incorporated into 
the “Summary of Enterprise Penetration” section (slides 7-8). It is not 
necessary to divide the CIs by center in this table since they are divided in that 
manner on the subsequent funding chart (slide 9). This would eliminate some 
of the redundancies in these introductory slides and move the presentation 
more quickly to the detailed information that Dr. Greenfield would like to see. 

Furthermore, it is suggested that it be made clear that Software Assurance is 
an overall theme for all CIs. 
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2.2 Questions Posed by Dr. Greenfield 


Question 

Topic & Slide 


How many years do CIs 
usually take? 

Summary of Center 
Initiatives (5) 

SY responded that CIs are 
usually one to three years with 
an average duration of two years. 

Did we work with the CIO 
office on the NPD 2820 
Metrics effort? 

Summary of Enterprise 
Penetration (7) 

SY responded that Ted Hammer 
from Goddard Space Flight 
Center worked on this CL 

What is the money allocated 
to Headquarters? 

FY99 Center Initiatives (9) 

SY indicated that this money was 
returned because it was “over 
guideline”)?) He also indicated 
that it came back to the IV&V 
Facility this year. 


3 WVU/IV&V Facility Research Program FY99-00 & IV&V Facility Center 
Initiatives 

Tim Menzies of West Virginia University presented the status of their research 
efforts and of CIs performed by IV&V Facility personnel. Dr. Greenfield 
expressed great interest in this portion of the presentation and responded 
favorably to the information and the presenter. 

3. 1 Issues 

Concern 

Dr. Greenfield expressed great interest in the application of the Quality 
Enabling Software Technologies (QuEST) toolkit to a NASA project. The 
principal investigator stated that such a case study would be occurring in the 
future. 

Analysis 

At the time of SETA’s assessment of the QuEST Cl, it appeared that QuEST 
was not an activity that the IV&V Facility was interested in pursuing due to 
some legal issues with the Microelectronics & Computer Technology 
Corporation (MCC). However, it now appears that the IV&V Facility is 
committed to conducting a case study. Since the Cl has been officially closed, 
there does not appear to be money for the study to be performed through Cl 
funding. 


1 SY=Siamak Yassini, LB=Dr. Louis Blazy, TM=Tim Menzies, CC=Cynthia Calhoun, KC=Ken 
Costello 
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Recommendation 

It is recommended that the Principal Investigator guide the case study using 

one of the following options: . 

1. The SETA Program conducted the assessment of the QuEST initiative, 
which required a familiarity with the toolkit. The SETA Program, therefore, 
has a background with the tool that positions them to be of value in 

conducting the case study. . 

2. QuEST is not unlike McCabe, Rational Rose, or any other such CASE tool. 

It is feasible that a student intern would be able to conduct the case study 
given appropriate learning time. However, it is uncertain if the status of the 
principal investigator as a student co-op would conflict with status as a 
student mentor, thus prohibiting this scenario. 


Concern . 

In several cases, there appears to be some confusion over what is being 
produced by CIs and what is being purchased. Examples of this include: 
QuEST and the Software Productivity Consortium (SPC). 

Analysis 

In both QuEST and the SPC CIs, it is stated in the slides that there are 
“products delivered.” During the SPC presentation by Tim Menzies, it was 
stated that “these are the products being produced.” This is an inaccurate 
statement made by a presenter who is not familiar, on a day-to-day basis, with 
that particular Cl. While it is true that “products” are being delivered, there is 
no way to tell from the presentation materials which products are developed 
and produced as a result of the Cl funding and other products which are 
purchased by the Cl funding. This leads to potential confusion for people who 
do not have a detailed understanding of each Cl (e.g. Dr. Greenfield and 
“substitute” presenters) . 

Recommendation 

It is recommended that a distinction be made, within the presentation packet, 
between products produced by Cl activities and products purchased by Cl 
funding. 
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3.2 Questions Posed by Dr. Greenfield 


Question 

Topic & Slide 

Summary of Response 

Is Software Risk Assessment 
Cl for risk assessment or 
identification? Is it useable 
now or in research stage? Is 
it a continuum? Is some 
being applied now? Does 
COCOMO give false risk 
identifications? 

Software Risk Assessment 
(16) 

Discussion with TM and LB 
indicated that this work is being 
continually refined as it is being 
used. As lessons are learned 
from the application process, 
they are incorporated back into 
the research to add additional 
focus. Dr. Greenfield indicated 
that he felt this was a balanced 
approach especially with the 
application of CARA analysis. 

Is ROI being done to advocate 
IV&V or to size the amount of 
IV&V for appropriate return? 
Are you working to 
understand how much money 
is worth investing in IV&V? 
What is your timeframe? 

Software Metrics & Return 
on Investment (ROI) (18) 

TM stated that the ROI work is 
not necessarily to advocate IV&V 
because the results could be 
negative. The work is being 
performed in order to know when 
the application of IV&V is cost 
effective. The timeframe for this 
effort is within the year. 

Is this to aid design to build 
higher reliability software? 
For new software? Is there a 
lot of data available on latent 
errors in new software 
releases? 

Software Reliability (19) 

TM responded that with all new 
software there is always a X” 
number of errors expected. This 
work could be applied to a COTS 
IV&V survey to predict latent 
errors, but that it needs to be 
fine-tuned to the particulars. 

Are we any closer to using 
the system? Are we 
dedicated to going to three 
string GPS? 

Specification-based 
Testing (23) 

LB and TM responded to 
questions in this area. The 
discussion indicated that $250K 
was being put into MAGR and 
that Averstar is working with 
information provided from WVU 
research. 


3.3 Action Items 
Action Item & Actionee 

Send Dr. Greenfield a copy of the Workshop on Risk Management (WoRM) 99 
workshop proceedings - Dr. Blazy 

Context & Recommended Action 

During the discussion on the Software Risk Assessment research work (slide 
16), Dr. Blazy mentioned WoRM 99. A brief discussion ensued on the success 
of the workshop and Dr. Greenfield indicated that he would like to obtain a 
complete copy of the proceedings. Dr. Blazy responded that he would provide 
them as soon as possible. It is recommended that the complete packet be sent 
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4 Center Initiatives from Other Centers 

Siamak Yassini presented information on all Cl efforts not being performed by 
IV&V Facility personnel. This area of the presentation raised the most 
questions and issues from Dr. Greenfield. Because the IV&V Facility manages 
these CIs “remotely,” this has been an area of consistent concern. 

4. 1 Issues 

Concern 

The quality review of Cl products was of great interest to Dr. Greenfield. It was 
suggested that rigorous review be conducted of all Cl products and that a 
“great deal” be returned to the authors in order to enforce a higher standard for 
the products that are delivered. 

Analysis 

Over the past year and a half, the IV&V Facility has begun to look more closely 
at the Cl products that are delivered. As products come in to the IV&V Facility 
they are being reviewed either by one of the NASA staff members or by a 
member of the SETA contract personnel. For the General Management Reviews 
(GMRs) and the OSMA Annual Review, the products that are reviewed by SETA 
personnel have been reported under SETA accomplishments. It is unclear 
where products reviewed by NASA personnel have been reported. This leads to 
inconsistencies in reporting for the deliverables and an incomplete picture of 
the work that the IV&V Facility’s personnel have been performing. 

Recommendation 

It is suggested that the IV&V Facility develop a methodology for tracking: (1) Cl 
products that are reviewed, (2) reviewers, and (3) accept/reject responses. This 
information should be included in the Cl information reported in each GMR 
and in the Annual Review. By putting the information within each Cl 
description, it will be easy for Dr. Greenfield to see which CIs have been 
producing acceptable products and who has been evaluating their success. 


Concern 

Technology transfer of products developed under CIs is a vital part of the Cl 
mission. This was expressed multiple times by Dr. Greenfield. It was indicated 
that he does not feel that this has been done effectively in the past and would 
like to ensure that products are being used by centers, not just developed. It 
was expressed that this is not a concern for the Jet Propulsion Laboratory (JPL) 
and Goddard Space Flight Center (GSFC) because they can often use their own 
developments. Furthermore, Dr. Greenfield expressed concern (during the 
Advanced Software- Fault Tolerance Strategies for Mission Critical Spacecraft 
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Applications presentation) about CIs not asking for follow on funding because 
he was concerned that their tool/developments would “get lost.” 

Analysis 

When CIs are proposed, it is required that they provide a description of the 
anticipated application of the technology / methodology within a NASA project. 
Unfortunately, there is no clear tracking mechanism to report progress for that 
technology transfer. In some cases, it is reported within the Cl status and in 
other cases it is reported within Cl deliverables as a pilot project 
documentation title. Because this is an area that is critical to Cl selection and 
is so important to Dr. Greenfield, it should be tracked more closely and 
consistently. 

Recommendation 

It is recommended that the IV&V Facility develop a plan for ensuring that the 
Cl products are indeed transferred to projects for use. This could be as simple 
as a tracking table that traces a product from delivery, through the review 
cycle, to implementation on a project, and that concludes with a brief 
implementation assessment or comments from the project manager who used 
the product. 

It is envisioned that this tracking table would be combined with the 
methodology for tracking the review of Cl products. 


Concern 

Dr. Greenfield expressed a need for a way to ensure that program managers 
were making the right decisions on funding allocation to ensure all areas of the 
project were covered (software, hardware, testing, research, etc). 

Analysis 

This is an industry-wide area of concern, not NASA specific. Therefore, it is 
expected that there is a body of work available to aid in the development of a 
cost scheduling/ allocation methodology that is NASA specific. 

The Software Engineering Institute (SEI) at Carnegie Mellon University (CMU) 
has developed the Personal Software Process (PSP) which includes, among 
other things, the use of Cost of Quality to manage quality and effectiveness. 

PSP measures appraisal, failure, and prevention costs. The cost measures that 
are included in these three areas are compiled from project costs in various 
areas. This type of existing work could be the foundation for a study to build 
the methodology that Dr. Greenfield is requesting. 

Recommendation 

It is recommended -that the IV&V Facility consider conducting further research 
in this area. A preliminary study of available information could be performed 
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through the SETA Program. Any gaps found or remaining information that is 
needed could then be developed through the SETA Program or through a Cl 
effort, depending on scope and available funding. 


Concern 

One concern that would potentially prevent future problems is ensuring that 
the language used on all slides is clear in detail. For example, slide 39 states 
“Support the V&V effort of the In-Situ Propellant Plant (ISPP) at Kennedy Space 
Center.” Dr. Greenfield questioned this and further discussion revealed that it 
is the plant to be placed on Mars. 

Analysis 

Ensuring that the information presented is unambiguous would prevent 
confusion for those referencing the slide presentation after the meeting and 
eliminate the need for such questions during the meeting. 

Recommendation 

It is suggested that during presentation dry runs, the audience include at lea§t 
one person who is not familiar with the projects in order to elicit questions that 
are “clarification” type issues. These are questions that slow the presentation, 
but can easily be answered through simple wording changes. 


4.2 Questions Posed by Dr. Greenfield 


Question 

Topic & Slide 

Summary of Response 

Are these metrics to be used? 
Are they good numbers? 

Engineering Reliability into 
SW Products Through 
Metrics (45) 

SY responded that the metrics 
are being used and that the 
numbers are good. 

Are we sending money to 
SATC? Do they have 
deliverables? Do they have 
funding from other sources? 
How many people are in 
SATC? Do we use JAVA on 
spacecraft? 

Software Assurance 
Technology Center (46) 

SY responded that we are 
sending money to SATC, but we 
are not their only source of 
funding. SATC does provide us 
with deliverables. There are 
approximately 79 people in 
SATC. During the discussion on 
Java, KC indicated that CLCS is 
using Java, but we have 
concerns about that since most 
space systems are using Ada 
and/or C++. 

Shouldn’t confidence in auto- 
code generators be a more 
significant thrust? Isn’t there 
a need for tools to give us 
added confidence outside of 
the development testing? 
What is the paradigm? 

Automatic Software Code 
Generator (61) 

LB indicated that this is an area 
of great interest for the IV&V 
Facility and that we are currently 
working on developing the 
paradigm to be used. 
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Question 

Topic & Slide 

Summary of Response 

Where did “Pete” come from? 

Software Planning Expert 
System - Ask Pete (76) 

SY responded that we were not 
certain on the origin of the name. 
Martha Wetherholt is the 
developer and it was her choice. 

Which IS? What is the 
current funding for IS? 

Cl Technical/ Management 
Issues (82) 

LB indicated that this was the IS 
effort from Ames. Steve 
Zornetzer has looked at providing 
some funding for the IV&V 
Facility depending on the 
amount that Ames receives. The 
level of funding for IS is still 
uncertain. Dr. Greenfield 
suggested that, in future 
negotiations, LB should request a 
percentage of the total funds 
rather than a flat funding 
amount. 


4.3 Action Items 
Action Item & Actionee 

Highlight interfaces between research and industry - Project Personnel 

Context & Recommended Action 

During the discussions on CIs, Dr. Greenfield repeatedly emphasized the 
importance of technology transfer to industry. It was directly requested, during 
the discussion on “Specification-based Testing,” that the IV&V Facility 
highlight the interfaces with industry because it is a key reason for facility 
funding. 

Section 2.1 of this document contains a description of the same concern in 
relation to the CIs. This action item expands that concern to cover WVU and 
NASA research activities as well. 

It is recommended that the Point of Contact (POC) for each project include, in 
their status, technology transferred from research to industry/NASA projects. 
The research efforts can be tracked on a table much like the one suggested for 
the Cl efforts. 


Action Item & Actionee 

Hold discussion with Dr. Greenfield on issue of balancing number of CIs with 
available budget (before May 2000) - Dr. Blazy 
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reflective of what our emphasis should be, per Dr. Greenfield. In his opinion, 
there should be a more prominent mention of “safety’’ as a mission of the IV&V 
Facility. 

Analysis 

Changing the mission and goals has presented a logistical problem in the past; 
there are several locations in which the updates must be made. However, the 
IV&V Facility is in the midst of updating the Operations Plan for the new year. 
This makes the timing appropriate for any necessary changes to the mission. 

Recommendation 

It is suggested that this statement be reworded as follows: “Increase software 
safety and quality, reduce software costs, and improve delivery time through 
the early detection and resolution of errors, and by utilizing and applying 
empirically based software engineering best practices.” 

This new phrasing should be applied in all instances where the IV&V Facility 
mission has been used. These include, but are not limited to: briefings, the 
IV&V Facility web sites, and the Code I web site (send to Gaye Graves at Ames). 


6.2 Questions Posed by Dr. Greenfield 

There were questions posed by Dr. Greenfield during the web demonstration, 
however they were not captured on paper (The presenter was also responsible 
for recording the meeting proceedings.). 


7 Summary 

Overall, the presentation was successful in that Dr. Greenfield appeared 
pleased and provided verbal praise for the IV&V Facility for having made great 
strides over the last year. Dr. Greenfield’s response to several IV&V Facility 
actions and suggestions was positive including: the hiring of Tom Mitchell of 
CMU as a consultant, the LaRC Process Verification (upon seeing results, Dr. 
Greenfield gave them an additional software person for staff), and the 
enthusiasm and expertise of several staff members and principal investigators. 

Dr. Greenfield continues to express confidence in the abilities of the IV&V 
Facility’s staff and looks for them to further enhance their management of 
activities receiving OSMA funding. To that end, it is critical that future 
briefings emphasize the technology transfer of products developed and provide 
evidence of Cl deliverable quality reviews. 
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SETA contract personnel noted the following overall issue for the presentation. 

Concern 

Changes in presentation duration and several other high priority events for the 
IV&V Facility resulted in a shortened presentation and contributed to errors in 
the presentation. Specific instances are as follows: 

1. Instead of 96 slides for a three-hour presentation, there were 96 slides for a 
two hour presentation. 

2. Changes were requested within the SETA section of the presentation. These 
changes were not incorporated. (Reference October 21, 1999 e-mail from 
Cynthia Calhoun) 

3. Multiple format errors throughout the presentation (i.e. font (slide 7), page 
number overlap on table (slide 5-7), acronyms not spelled out (multiple 
slides), text spacing (slide 65), title spacing (slide 30 & 65), partial full 
justification (slide 80), use of IWF for IV&V Facility (slides 92-95)). 

4. A statement was made that the SAIC Risk Cube methodology was used on 
the Boeing 757 independent assessment for Langley Research Center 
(LaRC). While it is true that the Risk Cube was initially used, it should be 
clarified that the assessment that was delivered to the customer did not 
employ the Risk Cube methodology. Due to the proprietary information 
issue, the independent assessment results using the Risk Cube were 
omitted and results from the IV&V Facility’s Criticality Analysis and Risk 
Assessment (CARA) method were incorporated into the final report that was 
delivered to the customer. 

5. Use of Lewis Research Center instead of Glenn Research Center (e.g. slides 
73-78). 

Analysis 

Many of these instances could be eliminated through a thorough review of the 
complete presentation after all changes have been incorporated. However, the 
errors during the presentation can largely be attributed to the change in 
meeting duration, which resulted in a rushed briefing. 

Recommendation 

It is recommended that future presentations are developed with key slides and 
“back-up” slides so presenters have advanced awareness of which slides are 
critical to their presentations. This would mitigate the need to make real-time 
decisions concerning which materials can be omitted or “glossed over.” 

Ideally, major changes to the presentation structure and content should not be 
made in the days immediately prior to the actual briefing. These decisions 
should be made and finalized well ahead of the briefing date. This would allow 
sufficient time to detect and fix all the minor formatting, spacing, and 
typographical errors, that inevitably result when many people are involved in 
contributing slides to a large presentation. 
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